2. Kullback-Leibler divergence

#math/KL-divergence

1. Definition

KL divergence is an important way to evaluate the similarity of 2 distributions, then give a reasonable Loss. Thus it is often used as the learning function of generated network, for example, VAE (Variational autoencoder)

(1) Concepts of KL-divergence

The Kullback-Leibler divergence(KL divergence)^[1] is also called relative entropy or I-divergence, is a type of statistical or distribution distance. which can measure one distribution $Q$ is different from true probability $P$.

$$\begin{array}{}\text{(1.1)}& \boxed{{\displaystyle D_{KL}(P||Q)=\sum _{x\in \mathcal{X}}P(x){\log }_{}\frac{P(x)}{Q(x)}}}\end{array}$$

or equivalent to :

$$\begin{array}{}\text{(1.2)}& \boxed{{\displaystyle D_{KL}(P||Q)=-\sum _{x\in \mathcal{X}}P(x){\log }_{}\frac{Q(x)}{P(x)}}}\end{array}$$

Also it is  expectation of the logarithmic difference between the probabilities $P$ and $Q$ using the probabilities $P$.

For continuous function, it is also called relative entropy (defined as integral):

$$\begin{array}{}\text{(1.3)}& D_{KL}(P||Q)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}p(x){\log }_{}\frac{P(x)}{Q(x)}=E_{x\sim p}[{\log }_{}\frac{P(x)}{Q(x)}]\end{array}$$

Note $D_{KL}$ always $\ge 0$

(2) Design Thoughts of KL-divergence

In the application purpose, we want the $q$ to approximate the distribution of $p$, So, we care about that : The q should be high where p is high :

Since we care more about the q is similar to p

Intuitively, there are three cases of importance^[2] (changed some expressions for consistency): • If $p$ is high and $q$ is high, then we are happy (i.e. low KL divergence). • If $p$ is high and $q$ is low then we pay a price (i.e. high KL divergence). • If $p$ is low, then we don't care (i.e. also low KL divergence, regardless of q).

---

1. https://en.wikipedia.org/wiki/Kullback–Leibler_divergence ↩︎

2. https://www.cs.cmu.edu/~epxing/Class/10708-17/notes-17/10708-scribe-lecture13.pdf ↩︎